A relational database is a collection of related data that can be represented by two-dimensional tables of columns and rows wherein information can be derived by performing set operations on the tables, such as join, sort, merge, and so on. The data stored in a relational database is typically accessed by way of a user-defined query that is constructed in a query language such as Structured Query Language (SQL).
Often it is useful to extract data from one or more sources, transform the data into some more useful form, and then load the results to a separate destination. A data warehouse, for example, is a central repository for all or significant parts of the data that an entity's various business systems collect and store (often in separate databases), the purpose of the data warehouse being to support data mining, decision support systems (DSS), and other data actions. Data from various sources is selectively extracted and organized on the data warehouse database for use by analytical applications and user queries. Data warehousing emphasizes the capture of data from diverse sources for useful analysis and access.
In the context of a data warehousing, and more generally for managing databases, extract-transform-load (ETL) refers to three separate functions of obtaining, processing, and storing data. The extract function reads data from a specified source database and extracts a desired subset of data. The transform function works with the acquired data—using rules or lookup tables, or creating combinations with other data—to convert it to the desired state as defined by the specific ETL tool. The load function is used to write the resulting data (either all of the subset or just the changes) to a destination database. Various and diverse ETL tools can be used for many purposes, including populating a data warehouse, converting a database of a specific type into a database of another type, or migrating data from one database to another. However, when transforming data, a copy of the data extracted from the source and resident in memory (the primary copy) is again copied to a separate memory location, manipulated, and then copied back to the primary copy location. This unnecessary copying is both inefficient and can lead to data inconsistencies. What is needed in the art is a means for directly transforming a single buffered copy of data in memory without further copying of said data.